Performance Characterisation in Computer Vision: A Guide to Best Practices

It is frequently remarked that designers of computer vision algorithms and systems cannot reliably predict how algorithms will respond to new problems. A variety of reasons have been given for this situation, and a variety of remedies prescribed [99, 213, 33]. Most of these involve, in some way, paying greater attention to the domain of the problem and to performing detailed empirical analysis. This document is not intended to be a a standard review article. Rather, the goal of this paper is to review only what we see as current best practices in these areas and also suggest refinements that may benefit the field of computer vision. A distinction is made between the historical emphasis on algorithmic novelty and the increasing importance of validation on particular data sets and problems. 1With the exception of Bill Crum who was invited to contribute later.